Computer-Assisted Keyword and Document Set Discovery from Unstructured Text

The (unheralded) first step in many applications of automated text analysis involves selecting keywords to choose documents from a large text corpus for further study. Although all substantive results depend crucially on this choice, researchers typically pick keywords in ad hoc ways, given the lack of formal statistical methods to help. Paradoxically, this often means that the validity of the most sophisticated text analysis methods depends in practice on the inadequate keyword counting or matching methods they are designed to replace. The same ad hoc keyword selection process is also used in many other areas, such as following conversations that rapidly innovate language to evade authorities, seek political advantage, or express creativity; generic web searching; eDiscovery; look-alike modeling; intelligence analysis; and sentiment and topic analysis. We develop a computer-assisted (as opposed to fully automated) statistical approach that suggests keywords from available text, without needing any structured data as inputs. This framing poses the statistical problem in a new way, which leads to a widely applicable algorithm. Our specific approach is based on training classifiers, extracting information from (rather than correcting) their mistakes, and then summarizing results with Boolean search strings. We illustrate how the technique works with examples in English and Chinese. ∗Our thanks to Dan Gilbert for helpful suggestions. †Institute for Quantitative Social Science, 1737 Cambridge Street, Harvard University, Cambridge MA 02138; GKing.harvard.edu, [email protected], (617) 500-7570. ‡Institute for Quantitative Social Science, 1737 Cambridge Street, Harvard University, Cambridge, MA 02138; www.patricklam.org §Department of Government, 1737 Cambridge Street, Harvard University, Cambridge MA 02138; scholar.harvard.edu/mroberts